Consistent Weighted Sampling Made Fast, Small, and Easy

Document sketching using Jaccard similarity has been a workable effective technique in reducing nearduplicates in Web page and image search results, and has also proven useful in file system synchronization, compression and learning applications [6, 4, 5]. Min-wise sampling can be used to derive an unbiased estimator for Jaccard similarity and taking a few hundred independent consistent samples leads to compact sketches which provide good estimates of pairwise-similarity. Early sketching papers handled weighted similarity, for integer weights, by transforming an element of weight w into w elements of unit weight, each requiring their own hash function evaluation in the consistent sampling. Subsequent work [12, 19, 14] removed the integer weight restriction, and showed how to produce samples using a constant number of hash evaluations for any element, independent of its weight. Another drastic speedup for sketch computations was given by Li, Owen and Zhang [17] who showed how to compute such (near-)independent samples in one shot, requiring only a constant number of hash function evaluations per element. Unfortunately this latter improvement works only for the unweighted case. In this paper we give a simple, fast and accurate procedure which reduces weighted sets to unweighted sets with small impact on the Jaccard similarity. This leads to compact sketches consisting of many (near-)independent weighted samples which can be computed with just a small constant number of hash function evaluations per weighted element. The size of the produced unweighted set is furthermore a tunable parameter which enables us to run the unweighted scheme from [17] in the regime where it is most efficient. Even when the sets involved are unweighted, our approach gives a simple solution to the densification problem that [20, 21] attempt to address. Unlike previously known schemes, ours does not result in an unbiased estimator. However, we prove that the bias introduced by our reduction is negligible and that the standard deviation is comparable to the unweighted case. We also empirically evaluate our scheme and show that it gives significant gains in computational efficiency, without any measurable loss in accuracy.